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earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)13 Responsive to communication(s) filed on 31 March 2004 . 
2a)n This action is FINAL. 2b)ISI This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 
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4) IEI Claim(s) 7-25 is/are pending in the application. 
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Art Unit: 2624 

DETAILED ACTION 
Election/Restrictions 

1 . Restriction to one of the following inventions is required under 35 U.S. C. 
121: 

I. Claims 1-19, drawn to " performing a spatio-temperol segmentation 
analysis on the video", classified in class 382, subclass 173. 

II. Claims 20-25, drawn to "determining a set of volumetric objects by 
means shift video segmentation with respect to keyframes 
interpolation", classified in class 348, subclass 36. 

The inventions are distinct, each from the other because of the following reasons: 

2. Inventions I and II are related as subcombinations disclosed as usable 
together in a single combination. The subcombinations are distinct if they do not 
overlap in scope and are not obvious variants, and if it is shown that at least one 
subcombination is separately usable. In the instant case, subcombination II has 
separate utility such as determining a stroke sheet that is lying within the 
semantic region in order to provides a very powerful system for the stylization of 
video. See MP BP § 806.05(d). 

The examiner has required restriction between subcombinations usable 
together. Where applicant elects a subcombination and claims thereto are 
subsequently found allowable, any claim(s) depending from or otherwise 
requiring all the limitations of the allowable subcombination will be examined for 
patentability in accordance with 37 CFR 1.104. See MPEP § 821.04(a). 
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Applicant is advised that if any claim presented in a continuation or divisional 
application is anticipated by, or includes all the limitations of, a claim that is 
allowable in the present application, such claim may be subject to provisional 
statutory and/or nonstatutory double patenting rejections over the claims of the 
instant application. 

3. Because these inventions are independent or distinct for the reasons 
given above and there would be a serious burden on the examiner if restriction is 
not required because the inventions have acquired a separate status in the art in 
view of their different classification, restriction for examination purposes as 
indicated is proper and there would be a serious burden on the examiner if 
restriction is not required because the inventions require a different field of 
search (see MPEP § 808.02), restriction for examination purposes as indicated is 
proper. 

4. During a telephone conversation with Mr.Rustan Hill on 04/24/2007 a 
provisional election was made without traverse to prosecute the invention of 
"stylization of video by performing spatio-temporal segmentation analysis", claims 
1-19. Affirmation of this election must be made by applicant in i-eplying to this 
Office action. Claims 20-25 are withdrawn from further consideration by the 
examiner, 37 CFR 1.142(b), as being drawn to a non-elected invention. 

Examiner Notes 

5. Examiner cites particular columns and line numbers in the references as 
applied to the claims below for the convenience of the applicant. Although the 
specified citations are representative of the teachings in the art and are applied 
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to the specific limitations within the individual claim, other passages and figures 
may apply as well. It is respectfully requested that, in preparing responses, the 
applicant fully consider the references in entirety as potentially teaching all or 
part of the claimed invention, as well as the context of the passage as taught by 
the prior art or disclosed by the examiner 

Claim Rejections - 35 USC § 103 

6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for 
all obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described 
as set forth in section 102 of this title, if the differences between the subject matter sought to 
be patented and the prior art are such that the subject matter as a whole would have been 
obvious at the time the invention was made to a person having ordinary skill in the art to which 
said subject matter pertains. Patentability shall not be negatived by the manner in which the 
invention was made. 

7. Claims 1-19 are rejected under 35 U.S.C. 103(a) as being unpatentable by 
Mittal et a! (US 2005/0286764) in view of DeMenthon et al (Spatio-Temporal 
Segmentation of Video by Hierarchical Mean Shift Analysis, July 2002) 

As to claim 1 , Mittal teaches a method for stylizing video, comprising: 
performing a spatio-temporal segmentation analysis on the video to identify three 
dimensional volumes (optical flow computation, spatio-temporal domain, 
paragraph [0012]) of contiguous pixels having a similar characteristic; receiving 
an input identifying a group of the three dimensional volumes; and identifying the 
group of three dimensional volumes as a single semantic region (forming a 
background model in a high-dimensional space, paragraph [0019]). Note that the 
optical flow constraint equation is applied at a given point defined by a spatial 
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location and time to obtain respective constraints; applying an error function to 
combine the respective constraints from each the given point within a defined 
region for deriving a characteristic function; deriving a motion estimate from the 
characteristic function; jand comparing the motion estimate with a given 
uncertainty model so as to derive a figure of uncertainty for optical flow 
measurement data, paragraph [0022], paragraph [0078]). 
While Mittal et al meets a number of the limitations of the claimed invention, as 
pointed out more fully above, Mittal fails to specifically teach the spatio-temporal 
segmentation by mean shift analysis. Specifically, DeMenthon et al. teaches the 
use of spatio temporal segmentation of video sequences by adopting a 
hierarchical clustering method, which operates by repeatedly applying mean shift 
analysis over increasingly high ranges. Because the segmentation analysis of 
DeMenthon includes color segmentation and motion segmentation, as well as 
consitent labeling of regions over time which amounts to region tracking. It would 
have been obvious to one of ordinary skill in the art to use the spatio-temporal 
segmentation analysis in Mittal in order to provide a hierarchical segmentation of 
data thus obtaining cleaner boundaries and help in video indexing and retrieval. 
Therefore, the claimed invention would have been obvious to one of ordinary skill 
in the art at the time of the invention by applicant. 

As to claim 2, Mittal teaches the method of claim 1 , wherein the similar 
characteristic comprises color (figure 9, color representation, note that the 
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dynamic scene comprises utilizing optical flow information to discriminate 
between observations exhibiting similar color components) 

As to claim 3, Mittal teaches the method of claim 1 , wherein the spatio- 
temporal segmentation analysis comprises an anisotropic kernel mean shift 
segmentation procedure (anisotropic non-linear filters can be considered, 
paragraph [0129]) 

As to claim 4,Mittal teaches the method of daim 1 , wherein the input 
comprises an interactive user input (the user determines the degree of sensitivity, 
paragraph [0188]). 

• As to claim 5, Mittal teaches the method of daim 1 , wherein the three 
dimensional volumes of contiguous pixels comprise segments (apparatus for 
dividing the image into blocks, paragraph [0026]). 

As to claim 6,Mittal teaches the method of daim 5, wherein the user input 
comprises outlining a plurality of segments (apparatus for dividing the image into 
blocks, paragraph [0026]). 

As to claim 7, teaches the method of claim 6, wherein the outlining 
(morphological operation) is performed on a number of keyframes of the video, 
the number of keyframes being fewer than a total number of frames of the video 
(note that the image is divided into blocks, and for each block, PCA is performed 
using a certain number of previous frames, paragraph [0026-0028], figure 1) 

As to claim 8, teaches the method of claim 7, wherein additional segments 
on frames of the video other than keyframes are identified by determining a 
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relationship of the additional segments to the segments outlined on the 
keyframes, (figures 9-11; paragraph [0079], 

As to claim 9 and 10, Mittal teaches relationship comprises at least a 
portion of the additional segments being enclosed by one or more of the 
segments outlined on the keyframes and at least a portion comprises at least a 
majority of pixels of the additional segments (The method is able to capture the 
long term dynamic characteristics of the scene, temporal and structural 
relationships between different pixels and multiple modalities of dynamic 
behavior, paragraph [0109], [0110]). 

As to claim 1 1 , teaches the method of claim 1 , further comprising applying 
a stylization to the single semantic region (note that a simple model keeps a 
single background image, paragraph [0046]). 

As to claim 12, Mittal teaches the method of claim 1 1 , wherein the 
stylization comprises a mean shift technique (The Variable Bandwidth Mean shift 
and Data-Driven Scale Selection, paragraph [0082]) 

Claims 13-19 differ from claims 1-12 only in that claims 1319 are computer 
claim whereas, claims 1-12 are methods claim. Thus, claims 13-19 are analyzed 
as previously discussed with respect to claims 1-12 above. 

Conclusion 

8. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 
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Ziao et al (US 2007/0092122 ) Is cited to teach estimation of optical flow 
by performing color segmentation and adaptive bilateral filtering to regularize the 
flow field to achieve a more accurate flow field estimation. After creating pyramid 
models for two input image frames, color segmentation is performed. Next, 
starting from a top level of the pyramid, additive flow vectors are iteratively 
estimated between the reference frames by a process including occlusion 
detection, wherein the symmetric property of backward and forvyard flow is 
enforced for the non-occluded regions. Next, a final estimated optical flow field is 
generated by expanding the current pyramid level to the next lower level and the 
repeating the process until the lowest level is reached. This approach not only 
generates efficient spatial-coherent flow fields, but also accurately locates flow 
discontinuities along the motion boundaries. 

Cham et al (US 6,795567) is cited to teach tracking an object model in a 
sequence of frames where the object model comprises a plurality of features and 
is described by a model state, includes both selecting an unregistered feature of 
the object model and selecting an available frame from the sequence of firames, 
to minimize a cost function of a subsequent search. A search is performed for a 
match of the selected model feature to the image in the selected frame in order 
to register the feature in that frame. The model state is then updated for each 
available frame. The steps of selecting, searching and updating are repeated. 
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9. Any inquiry concerning this communication or earlier communications from 
the examiner should be directed to Nancy Bitar whose telephone number is 571 - 
270-1041. The examiner can normally be reached on Mon-Fri (7:30a.m. to 
5:00pm). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Joseph Mancuso can be reached on 571-272-7695. The fax phone 
number for the organization where this application or proceeding is assigned is 
571-273-8300. ' 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR 
only. For more information about the PAIR system, see http://pair- 
direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll- 
free). If you would like assistance from a USPTO Customer Service 
Representative or access to the automated information system, call 800-786- 
9199 (IN USA OR CANADA) or 571-272-1000. y 



Nancy Bitar 




04/23/2007 



